® Check for updates 


Original Research 


The Journal of School Nursing 
1-10 

© The Author(s) 2020 

Article reuse guidelines: 
sagepub.com/journals-permissions 
DOI: 10.1177/1059840520925522 
journals.sagepub.com/home/jsn 


@SAGE 


Intervention Adherence and Self-Efficacy 
as Predictors of Child Outcomes in School 
Nurse—Delivered Interventions for 
Anxiety 


EB Caron, PhD' ©, Kelly L. Drake, PhD”?, Catherine E. Stewart, PhD‘, 
Michela A. Muggeo, PsyD*, and Golda S. Ginsburg, PhD® 


Abstract 


This study examined the association between two implementation factors, nurse-reported intervention adherence and self- 
efficacy, and children’s outcomes in school nurse—delivered anxiety interventions. Data were collected in a pilot randomized 
controlled effectiveness trial with 54 children and 21 school nurses. Nurses implemented either a cognitive behavioral or 
relaxation-skills-only intervention. Nurse questionnaires assessed implementation factors. Independent evaluators assessed 
changes in children’s anxiety symptoms at postintervention and at 3-month follow-up using clinical improvement and global 
functioning scales. Regression analyses indicated that greater intervention adherence was associated with greater anxiety 
symptom improvement at follow-up. Nurse self-efficacy interacted with intervention group, such that nurses with higher self- 
efficacy who implemented the cognitive behavioral intervention tended to have children show improvement and higher 
postintervention functioning. The impact of implementation factors on children’s outcomes may differ depending on inter- 
vention type. Self-efficacy may be important for nurses using relatively complex interventions. Intervention adherence should 


be supported through training and consultation. 
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Excessive anxiety is prevalent and affects more than 20% 
of youth at some point during childhood or adolescence 
(Copeland et al., 2014). Symptoms of excessive anxiety 
include maladaptive thoughts (e.g., youth with social 
anxiety think everyone will laugh at them; youth with 
separation anxiety worry about something dangerous hap- 
pening to their parents), somatic symptoms (e.g., sto- 
machaches, headaches), and avoidant behavior (e.g., 
youth with social anxiety may refuse to speak in class; 
youth with separation anxiety may avoid going to 
school). Excessive anxiety is impairing and negatively 
impacts children’s school performance, friendships, and 
family functioning (Swan & Kendall, 2016). Fortunately, 
psychosocial treatment for excessive anxiety is effective; 
specifically, cognitive behavioral therapy (CBT), which 
focuses on changing children’s maladaptive thoughts and 
reducing somatic symptoms and avoidant behavior 
related to anxiety, has been established as an evidence- 
based treatment (James et al., 2015). Meta-analyses have 
shown that CBT is effective in reducing children’s anxi- 
ety (James et al., 2015). 


However, getting CBT to the anxious children who could 
benefit from it is difficult. Only about one third of children 
with excessive anxiety receive treatment (Chavira et al., 
2004; Merikangas et al., 2010, 2011). Barriers to children’s 
access include expense, difficulty finding a service provider, 
attending weekly therapy appointments that interfere with 


' Department of Psychological Science, Fitchburg State University, MA, 
USA 

? Department of Psychiatry and Behavioral Sciences, Johns Hopkins Uni- 
versity, Baltimore, MD, USA 

3 Anxiety Treatment Center of Maryland, Columbia, MD, USA 

* Department of Psychological Sciences, University of Connecticut, Storrs, 
CT, USA 

5 Clarus Health Alliance, Norwich, CT, USA 

® Department of Psychiatry, University of Connecticut School of Medicine, 
West Hartford, CT, USA 


Corresponding Author: 

Golda S. Ginsburg, PhD, Department of Psychiatry, University of 
Connecticut School of Medicine, 65 Kane Street, West Hartford, CT 
06119, USA. 

Email: gginsburg@uchc.edu 


The Journal of School Nursing XX(X) 


family schedules, and the stigma of seeking mental health 
treatment (Salloum et al., 2016). One solution to these 
barriers is the provision of school-based mental health 
services (Weist et al., 2017). Given that school psychol- 
ogists and counselors may be burdened by other priorities 
(Weist et al., 2017), and school nurses may be familiar 
with anxious students because of their somatic com- 
plaints, “task shifting” mental health interventions for 
anxiety to school nurses offers a promising opportunity 
(Hoeft et al., 2018). 

School nurses have been involved in mental health care 
for decades; in fact, in 2002—2003, school nurses reported 
spending 32% of their time providing mental health services 
(Foster et al., 2005). However, there has been relatively little 
research on school nurses’ use of evidence-based mental 
health services for anxiety. An early exception, Stallard 
et al. (2007, 2008) found that a school nurse—delivered uni- 
versal preventative intervention reduced children’s anxiety 
levels. To our knowledge, the only research on school 
nurse—delivered evidence-based interventions for children 
diagnosed with anxiety disorders has focused on a modular 
cognitive behavioral intervention, Child Anxiety Learning 
Modules (CALM; Ginsburg et al., 2019; Muggeo et al., 
2017). In contrast to this lack of research, however, a large 
majority of school nurses (94%) report the desire to receive 
training in evidence-based anxiety-reduction skills (Muggeo 
& Ginsburg, 2019). 

This need for training can be addressed through imple- 
mentation science, the study and practice of integrating 
evidence-based treatments into community care (Fixsen 
et al., 2005). Given the challenges of implementing 
evidence-based treatments in schools (e.g., Hulleman & 
Cordray, 2009), examination of implementation factors that 
influence child outcomes is needed. Implementation factors 
are best understood in the context of an implementation 
framework (Tabak et al., 2012). The Consolidated Frame- 
work for Implementation Research integrated a number of 
existing implementation frameworks (Damschroder et al., 
2009). It proposed that implementation factors exist within 
five major domains, including characteristics of the inter- 
vention, the inner setting, the outer setting, the implementa- 
tion process, and the providers. These implementation 
factors interact to influence implementation effectiveness 
in complex ways (Damschroder et al., 2009). The current 
study focuses on one of these domains: characteristics of 
providers. Understanding whether and how provider-level 
implementation factors predict client outcomes would allow 
implementation processes to target these factors in order to 
improve outcomes (Fixsen et al., 2005). Intervention adher- 
ence and provider self-efficacy are two important implemen- 
tation factors. 

Adherence is a component of treatment fidelity and rep- 
resents the extent to which interventions are administered as 
designed (Perepletchikova & Kazdin, 2005). Lowered treat- 
ment fidelity has been implicated in the reduced effects of 


evidence-based treatments in community-based settings 
(Hulleman & Cordray, 2009). However, findings linking 
adherence with treatment outcomes have been mixed both 
across the wider field of evidence-based treatments for men- 
tal health (Breitenstein et al., 2010) and within youth anxiety 
treatment research (e.g., Bjaastad et al., 2018; Liber et al., 
2010). Given the focus on adherence in training and con- 
sultation, more research is needed. 

Self-efficacy, or a provider’s belief in their ability to 
achieve implementation goals, is theorized to be a critical 
factor in individual behavior change and implementation 
outcomes (Damschroder et al., 2009). Prior work has estab- 
lished a link between providers’ self-efficacy and their fre- 
quency of evidence-based treatment use after training 
(Klimes-Dougan et al., 2009; Turner et al., 2011). For exam- 
ple, among primary care providers (61% nurses), providers’ 
self-efficacy predicted how often they used the Primary Care 
Triple P—Positive Parenting Program—in their usual prac- 
tice (Turner et al., 2011). Similarly, among school-based 
counselors, providers’ self-efficacy significantly predicted 
greater knowledge of evidence-based treatments for anxiety 
and predicted usage of evidence-based treatments for anxi- 
ety at the trend level (p = .069; Schiele et al., 2014). How- 
ever, it is less clear whether self-efficacy relates to 
adherence and competence (Klimes-Dougan et al., 2009), 
and research on the impact of self-efficacy on client treat- 
ment outcomes is needed. 

The current study investigated the impact of self-reported 
adherence and self-efficacy on children’s outcomes after 
receiving one of two brief interventions for pediatric anxiety 
implemented by school nurses. One intervention (CALM) 
was a modular cognitive behavioral—based intervention and 
the other (CALM-Relaxation only [CALM-R]) focused on 
relaxation strategies only. The current study was conducted 
in the context of a pilot randomized controlled trial, which 
found that both interventions resulted in significant reduc- 
tions in anxiety (Ginsburg et al., 2019). Although the core 
elements of the two interventions represent mature, well- 
established components of CBT, and similar modular CBT 
approaches to treat and prevent anxiety previously have 
been used by school nurses and counselors (Ginsburg 
et al., 2012; Stallard et al., 2007), this pilot trial was the first 
test of these specific modular interventions for anxiety deliv- 
ered by school nurses. 

We hypothesized that higher levels of adherence and self- 
efficacy would be associated with better child outcomes 
(i.e., lower anxiety and higher functioning), regardless of 
intervention condition. We also explored whether associa- 
tions depended on the type of intervention being implemen- 
ted, as intervention characteristics such as complexity may 
impact implementation (Damschroder et al., 2009). We 
expected that findings would highlight provider-level imple- 
mentation factors that could be targeted to improve chil- 
dren’s outcomes. 
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Method 


Data were collected as part of a pilot randomized controlled 
trial using a cluster-randomized design to evaluate the fea- 
sibility and preliminary effectiveness of CALM. CALM-R 
was developed as an active control condition. The study 
protocol was approved by the institutional review board, and 
participants including nurses, children, and parents provided 
informed assent/consent for their participation. 


Nurse Participants 


Nurse participants included 8 CALM and 13 CALM-R 
nurses in 21 public schools in two eastern states in the 
United States. This sample reflects 70% of nurses who con- 
sented to the study; 8 nurses were trained in CALM and 
CALM-R but did not treat any children and 1 additional 
nurse dropped out of the study before implementing the 
intervention, and these nurses were not included in the cur- 
rent study. All nurses were female. The majority of nurses 
(n = 18, 86%) identified as White and non-Hispanic, two 
identified as Hispanic, and one as American Indian/Alaska 
Native. On average, nurses were 52 (SD = 7) years old and 
had been working as school nurses for 10 (SD = 7) years. 
Eight (38%) had an associate degree, 10 (48%) had a bache- 
lor’s degree, and 3 (14%) had a master’s degree, and all 
nurses were licensed in their state. On average, nurses imple- 
mented the intervention with 2.5 children (SD = 1.5, range: 
1-6) and covered 5.8 modules (SD = 0.8, range: 1-6) over 
the course of 6.8 weeks (SD = 2.2, range: 0-15) in 2016- 
2017. There were no demographic differences between 
nurses assigned to CALM and CALM-R. 


Child Participants 


Child participants included 20 children assigned to CALM 
and 34 assigned to CALM-R (61% of those referred). The 
majority of children (1 = 37, 69%) were female, and on 
average, they were 8.3 (SD = 2.0) years old. Most children 
(n = 37, 69%) were White/non-Hispanic, 13 (24%) were 
Hispanic, 3 (6%) were multiracial, and 1 (2%) was Black. 
Fifty-seven percent of the sample reported family incomes 
of over US$80,000 per year, while 22% of the sample 
reported receiving income-based free school lunch. There 
were no demographic differences between children assigned 
to CALM and CALM-R. Additional sample characteristics, 
CONSORT diagrams, and inclusion/exclusion criteria can 
be found in Ginsburg et al. (2019). 


Procedure 


Nurses working in public and private schools with youth 
aged 5—12 years were recruited through their district’s nur- 
sing supervisor, flyers in the school, and word of mouth. 
Sample size was based on grant-proposed enrollment of 20 
nurses and 60 children, with estimates of 80% power and 
large within-group intervention effect sizes. Nurses 


provided informed consent and were randomly assigned to 
CALM or CALM-R. They completed a 1-day training in 
their assigned intervention and were provided with interven- 
tion materials, including the intervention manual, training 
videos, and psychoeducational handouts. Nurses in both 
conditions were assigned a doctoral-level clinical psycholo- 
gist to provide optional consultation. They were not required 
to demonstrate a particular level of knowledge, adherence, 
or competence prior to beginning to implement the interven- 
tions, and no training certificates were provided. 

Children were recruited through participating school 
nurses who referred children who frequently visited the nurse 
with anxiety/somatic symptoms. Children were also referred 
through teachers who were aware of children’s anxiety symp- 
toms, by parents who saw flyers in the school newsletter, and 
through word of mouth. After providing assent/consent, chil- 
dren and parents completed a baseline evaluation in which 
independent evaluators completed diagnostic clinical inter- 
views and formulated ratings of anxiety symptom severity 
and global functioning. Children were assigned to the condi- 
tion in which the nurse in their school had been trained and 
therefore were not randomly assigned to condition. 

Once enrolled, children began meeting with their school 
nurse individually, usually in the nurse’s health office or an 
empty classroom. Both interventions included six modules 
that were designed to be delivered to children in brief meet- 
ings lasting 20-30 min. Meetings were intended to occur 
weekly during the school day. After each intervention ses- 
sion, nurses completed a session summary form (adapted 
from Becker et al., 2012), which included items assessing 
adherence and self-efficacy. No contextual changes in 
school nursing policy or staffing were noted that would have 
affected nurses’ intervention implementation, and no harms 
or unintended effects were reported. 

In postintervention and 3-month follow-up evaluations 
with children and their parents, independent evaluators 
again completed a diagnostic clinical interview and mea- 
sures of anxiety symptom severity and global functioning. 
They also rated children’s symptom improvement compared 
to the baseline evaluation. Independent evaluators had a 
master’s or doctoral degree in psychology and were trained 
to reliability through training tapes and observation and 
co-rating by an expert trainer. Once reliable, evaluators 
received supervision and feedback on each evaluation. Inde- 
pendent evaluators were unaware of children’s intervention 
group assignments, and over 80% indicated that they were 
“completely uncertain” of group assignment at post- and 
follow-up evaluations. 


Intervention Conditions 


CALM. The six CALM modules included core components of 
CBT for pediatric anxiety: (1) psychoeducation (i.e., provide 
child with information about nature of anxiety and introduce 
CBT model), (2) relaxation (1.e., identify child’s symptoms 
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of physiological tension and teach strategies to reduce ten- 
sion), (3) exposure (i.e., help child begin to face fears by 
putting themselves in feared situations), (4) cognitive 
restructuring (1.e., identify child’s negative self-talk and 
begin changing thoughts), (5) problem-solving (1.e., intro- 
duce and practice a problem-solving method), and (6) 
relapse prevention (i.e., review child’s progress and plan 
to use strategies in new fear-provoking situations). These 
components were designed to address three central manifes- 
tations of anxiety: cognitive symptoms (e.g., ruminative 
worry), somatic symptoms (e.g., physiological arousal), and 
behavioral symptoms (e.g., avoidance). CALM is described 
in greater detail in Drake et al. (2015). 


CALM-R. The six CALM-R modules focused on relaxation 
skills, with separate modules for (1) psychoeducation (i.e., 
provide child with information about anxiety and introduce 
concepts of physiological tension and relaxation), (2) deep 
breathing (i.e., introduce and practice method of slow, deep 
breathing), (3) progressive muscle relaxation (1.e., introduce 
and practice method of tensing and relaxing different muscle 
groups), (4) guided imagery (i.e., introduce and practice 
guided meditation), (5) individualized calming strategies 
(i.e., identify personal calming strategies such as going for 
a walk), and (6) relapse prevention (i.e., review child’s prog- 
ress and plan to use skills in new fear-provoking situations). 
Unlike CALM, CALM-R focused in depth on relaxation 
strategies and did not explicitly teach strategies to change 
anxious thoughts or behaviors. 


Measures 


Implementation factors 

Nurse-rated adherence. On each session summary form, 
nurses rated the item “Did you accomplish the goals of the 
module?” on a 7-point scale from 1 (did not accomplish 
goals) to 7 (accomplished all goals). Though this item was 
adapted from Becker et al. (2012), psychometric properties 
are not available, as in that study, the item was used to com- 
pare clinician ratings of goal accomplishment between differ- 
ent CBT modules, with findings of no differences. Ratings 
were averaged across all session summary forms available for 
each case. Internal consistency of nurses’ ratings across ses- 
sions was acceptable (Cronbach’s « = .75), indicating that 
while there was a good level of stability in individual nurses’ 
ratings, ratings also varied from session to session. 


Nurse-rated self-efficacy. On each session summary form, 
nurses also rated the item “How confident are you in your 
ability to implement CALM or CALM-R with this anxious 
student?” from 1 (not at all confident) to 7 (extremely con- 
fident). Ratings were averaged across all session summary 
forms available for each case. Internal consistency of nurses’ 
ratings across sessions was excellent (Cronbach’s « = .90), 
indicating that individual nurses’ self-efficacy was largely 
stable from session to session. 


Intervention outcomes 

Child symptom severity and improvement. Independent eva- 
luators used the Clinical Global Impression Scale—Severity 
(CGI-S) and Clinical Global Impression Scale—Improve- 
ment (CGI-I; Guy, 1976) to rate child symptom severity at 
all three time points and clinical improvement at post- and 
follow-up assessments. Independent evaluators’ ratings inte- 
grated information from parent and child interviews. The 
CGI-S and CGI-I are widely used in youth treatment 
research, with evidence of good interrater reliability and 
sensitivity to treatment change (e.g., Lang et al., 2016; Muf- 
son et al., 2004; Walkup et al., 2008). Severity scores range 
from 1 (normal; not at all ill) to 7 (extremely ill), and 
improvement scores range from | (very much improved) to 
7 (very much worse). Interrater agreement, defined as scor- 
ing within 1 point on the measure, was 100% for the CGI-S 
and 80% for the CGI-I. 


Child functioning. Child functioning was rated by indepen- 
dent evaluators using the Children’s Global Assessment 
Scale (CGAS; Shaffer et al., 1983). Independent evaluators’ 
ratings integrated information from parent and child inter- 
views. The CGAS has demonstrated acceptable interrater 
reliability, test-retest reliability, and sensitivity to treatment 
change (Green et al., 1994; Mufson et al., 2004; Walkup 
et al., 2008). Scores can range from | to 100, with higher 
scores reflecting better functioning across life domains. 
Interrater agreement in the current study, defined as scoring 
within the same decile, was 70%. 


Analyses 


Pearson’s correlations were used to examine bivariate 
associations among all study variables. Separate regres- 
sion analyses examined each implementation factor as a 
predictor of child outcomes, controlling for baseline child 
functioning/symptom severity and intervention group 
assignment. To examine whether associations between 
implementation factors and outcomes differed between 
intervention groups, additional regression analyses which 
included all previous predictors, as well as terms repre- 
senting the interaction between intervention group and 
implementation factor, were conducted. When an interac- 
tion term was significant, findings were probed to exam- 
ine the impact of the implementation factor within each 
group by running separate regressions for the CALM and 
CALM-R groups, including only the implementation fac- 
tor and baseline child functioning/symptom severity as 
predictors. 


Results 
Preliminary Analyses 


Bivariate correlations between variables are presented in 
Table 1. Baseline functioning (CGAS) and symptom sever- 
ity (CGI-S) were significant predictors of nurse self-efficacy 
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Table I. Means, Standard Deviations, and Correlations Between Variables in Full Sample. 

Variable Mean (SD) I 2 3 4 5 6 7 8 
|. BL CGAS 51.85 (5.70) — 

2. Post CGAS 57.89 (6.06) 64 — 

3. F-Up CGAS 60.61 (7.88) AR 82° _ 

4. BL CGI-S 4.41 (0.88) —.7 4° —.5 | #e* —.45** — 

5. Post CGI-I 2.91 (0.77) 10 —.5 | #e* —.37** —.23 — 

6. F-Up CGI-I 2.59 (0.79) 09 —.38** —.65*°* —.07 678 — 

7. Adherence 6.10 (0.84) ll AS 29* —.19 02 —.28 — 

8. Self-efficacy 5.70 (1.05) 35* 37 26 —.28* —.02 —.07 J56EF —_ 
9. Int. (CALM = 1) nia —.14 13 09 3 —.31* —.22 —.10 —.12 


Note. N = 54. Correlations are Pearson’s rs, with point-biserial correlations for intervention (a dichotomous variable). BL = baseline; F-Up = follow-up; 
CGAS = Children’s Global Assessment Scale; CGI-S = Clinical Global Impression Scale—Severity; CGI-I = Clinical Global Impression Scale-Improvement; Int. = 


Intervention; CALM = Child Anxiety Learning Modules. 
*p < 05. *p < 01. ***p < .001. 


Table 2. Partial Correlations From Regression Analyses Examining Implementation Factors as Predictors of Outcome. 


Independent Evaluator-Rated Child Outcomes 


Post Follow-Up 
CGAS CGI-l CGAS CGI-l 
Regressions examining nurse-rated predictors of outcomes 
Adherence 
Baseline severity (CGAS/ CGI-S) 63°" —.284 Agr —.14 
Intervention (CALM = 1) 254 —.244 18 —.22 
Adherence .10 —.07 27% —.33* 
Self-efficacy 
Baseline severity (CGAS/ CGI-S) BRE —.30* 40** —.10 
Intervention (CALM = 1) 254 —.254 17 —.20 
Self-efficacy 22 —.14 14 —.13 
Regressions examining interactions between nurse-rated predictors and intervention group 
Adherence 
Baseline severity (CGAS/CGI-S) 637°" —.244 A6** —.I1 
Adherence .03 .02 .09 —.22 
Intervention (CALM = 1) —.05 07 —.23 07 
Adherence x Intervention .08 —.10 .254 —.10 
Self-efficacy 
Baseline severity (CGAS/CGI-S) 66% —.22 AB —.03 
Self-efficacy —.05 07 —.08 02 
Intervention (CALM = |) —.39* .284 —.35* 22 
Self-Efficacy x Intervention 43°°* —.32* 38* —.25 


Note. Values in table represent partial correlations with significance levels identified. CGAS = Children’s Global Assessment Scale; CGI-S = Clinical Global 
Impression Scale—-Severity; CGI-| = Clinical Global Impression Scale-lmprovement; CALM = Child Anxiety Learning Modules. 


Xp < .10. *p < .05. **p < 01. ***p < 001. 


and child functioning at post- and follow-up assessments, so 
these variables were controlled for in the primary regression 
analyses. 


Primary Analyses 


Table 2 presents the results from regression analyses that 
examined nurse-reported intervention adherence and self- 
efficacy as predictors of child outcomes. The first set of 
analyses, presented in the top half of Table 2, examined 
the direct associations between nurse implementation 


factors and child outcomes. The second set of analyses, 
presented in the bottom of Table 2, included interaction 
terms to determine whether associations between imple- 
mentation factors and child outcomes differed between 
intervention groups. 


Adherence. Greater nurse-reported adherence to the interven- 
tions was associated with greater child anxiety symptom 
improvement (i.e., lower CGI-I scores), across groups, at the 
follow-up assessment. No interaction terms were significant, 
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Figure |. Effects of nurses’ self-efficacy on CGAS: Moderation by 
intervention group. Note. Dependent variables represent CGAS 
change (post/follow-up CGAS score—baseline CGAS score), as 
preintervention CGAS was a strong predictor of later CGAS 
scores. CGAS = Children’s Global Assessment Scale; CALM = 
Child Anxiety Learning Modules; CALM-R = Child Anxiety 
Learning Modules—Relaxation only. 
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Figure 2. Effects of nurses’ self-efficacy on CGI-I: Moderation by 
intervention group. Note. Lower values reflect greater improve- 
ment. Regression results depicted do not include Clinical Global 
Impression Scale—Severity as a control variable (unlike Table 2). 
CGI-I = Clinical Global Impression Scale-—lmprovement; CALM = 
Child Anxiety Learning Modules; CALM-R = Child Anxiety 
Learning Modules—Relaxation only. 


indicating that this association between adherence and out- 
come was consistent across intervention groups. There were 
no other significant associations between adherence and 
child outcomes. 


Self-efficacy. Self-efficacy was not a significant predictor of 
outcomes across intervention groups at the postintervention 
or follow-up evaluations. However, several interaction terms 
were significant. Specifically, at postintervention, as shown 
in Figures | and 2, CALM nurses who reported greater self- 
efficacy were more likely to have children show higher 
functioning on the CGAS and greater symptom improve- 
ment (i.e., lower scores) on the CGI-I. These interactions 
were supported by the results of probing regressions for 
each intervention condition, in which CALM nurses’ self- 
efficacy predicted children’s CGAS and CGI-I (Bs = .51 and 
—.60, both ps < .01), but CALM-R nurses’ self-efficacy did 
not (8 = —.06 and —.10, both ns). At follow-up, replicating 
postintervention findings, a significant interaction term sug- 
gested that CALM nurses who reported greater self-efficacy 
were more likely to have children show higher functioning 
on the CGAS. When probed with separate regressions for 
each intervention condition, as shown in Figure 1, CALM 
nurses’ self-efficacy predicted children’s CGAS (B = .58, 
p < .05), but CALM-R nurses’ self-efficacy did not 
(B = —.16, ns). 


Discussion 


The current study examined two provider-level implemen- 
tation factors in relation to children’s outcomes after two 
school nurse—administered brief interventions for child anxi- 
ety. Consistent with hypotheses and prior literature (Bjaas- 
tad et al., 2018; Ginsburg et al., 2012; Podell et al., 2013), 
nurse-reported adherence to the intervention modules for 
CALM and CALM-R was associated with greater improve- 
ment for children at the follow-up assessment. It is interest- 
ing that this association was found at the follow-up 
assessment and not at the postintervention assessment, but 
not without precedent, with similar results found by Bjaastad 
et al. (2018). It may have taken additional time for children 
to reap the benefits of high-quality intervention, or interven- 
tion quality may have been important for maintenance of 
gains once children were no longer receiving support. These 
findings suggest that implementation processes focused on 
increasing intervention adherence, including training and 
consultation, are likely to improve children’s outcomes, con- 
sistent with current recommendations (Damschroder et al., 
2009; Edmunds et al., 2013). 

Interestingly, findings regarding nurses’ self-efficacy dif- 
fered depending on which intervention nurses were imple- 
menting. These findings suggest the impact of another 
implementation factor, intervention complexity (Damschro- 
der et al., 2009). Specifically, for the more complex or multi- 
component intervention (CALM), nurses’ self-efficacy was 
linked to children’s outcomes, but for the simpler, more 
focused intervention (CALM-R), it was not. Self-efficacy 
did not differ between nurses in the CALM condition 
(M = 5.54, SD = 0.90) and the CALM-R intervention 
(M = 5.79, SD = 1.13), t(50) = .82, ns, so implementing 
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a more complex intervention did not appear to lower 
self-efficacy. Instead, it appears that less complex interven- 
tions can result in positive outcomes for children’s anxiety 
irrespective of how confident the provider feels about imple- 
menting the intervention. In contrast, with more complex 
interventions in which providers’ self-efficacy is linked to 
outcomes, it may be useful to assess and target providers’ 
self-efficacy in order to improve implementation outcomes. 

Implementation processes that are linked to increased 
provider self-efficacy include training, supervision, and a 
support network of peers who are also conducting the 
evidence-based treatment (Bohman et al., 2014; David & 
Schiff, 2017). The impact of these implementation processes 
was not assessed in the current study. Future research should 
examine how these implementation support processes can 
improve children’s outcomes through the mediator of 
increased provider self-efficacy. 

More broadly, these results suggest that in addition to 
acting directly as barriers or facilitators of implementation, 
characteristics of an intervention can also moderate associa- 
tions between implementation factors and intervention out- 
comes and may provide an explanation for why findings 
regarding implementation factors often differ across studies. 
For example, providers’ level of education has been posi- 
tively linked to implementation outcomes in some cases 
(e.g., Campbell et al., 2013) and negatively linked in other 
cases (e.g., Zvoch, 2009), most often failing to matter (e.g., 
Klimes-Dougan et al., 2009). These associations may be 
moderated by intervention complexity, with higher levels 
of education benefiting implementation of complex inter- 
ventions, but not mattering for simple interventions. In other 
cases, when an intervention is at odds with providers’ pre- 
vious training, higher levels of education may mean that 
previous training is more engrained and the new intervention 
is harder to adopt. This latter scenario reflects the interven- 
tion characteristic of compatibility. In addition to compat- 
ibility and complexity, a number of other intervention 
characteristics have been proposed to affect implementation 
outcomes, including flexibility, visibility, relative advantage 
over alternatives, and disruptiveness (Damschroder et al., 
2009; Grol et al., 2007). Future randomized controlled trials 
that include multiple interventions should continue to exam- 
ine moderation of implementation factor impact by interven- 
tion type to identify intervention characteristics that 
moderate expected associations. 


Strengths and Limitations 


The results should be taken in the context of several limita- 
tions of the study. First, adherence was self-reported by 
nurses, a method that is inexpensive and more feasible for 
implementation contexts but is likely less accurate and valid 
than observer-rated adherence (Breitenstein et al., 2010). 
The fact that self-reported adherence was associated with 
children’s outcomes despite this limitation is notable. In 


addition, although the implementation approach to the study 
(i.e., not requiring nurses to demonstrate a particular level of 
adherence or competence in delivering the intervention) 
increases generalizability of the results, this approach may 
have resulted in lower adherence and self-efficacy and 
affected children’s outcomes. Additionally, adherence and 
self-efficacy were assessed with single-item measures with 
limited evidence for validity and reliability, although aver- 
aging nurses’ ratings across multiple sessions appeared to 
increase the reliability of estimates in a way similar to that of 
a multi-item scale. Data were nested (i.e., children were 
nested within nurses), and statistical analyses did not address 
the nested structure of the data. Finally, the study’s small 
sample size limited power, and the specific nature of the 
sample (school nurses implementing mental health interven- 
tions) may limit generalizability of the results. Future 
research should attempt to replicate these findings in a larger 
sample, with observer-rated measures of treatment fidelity 
and multi-item scales with established psychometric prop- 
erties to assess self-reported constructs. 

The study also had several strengths. Nurses were ran- 
domly assigned to conditions, and randomization appeared 
successful, allowing us to infer that differences between the 
CALM and CALM-R groups could be attributed to some 
aspect of the intervention. As opposed to a treatment as usual 
comparison group that would include a diffuse mix of inter- 
vention strategies, both interventions were active treatment 
conditions that followed manualized protocols, limiting the 
possible explanations for the observed differences between 
intervention groups (e.g., to intervention complexity). In 
addition, children’s outcomes were assessed by independent 
evaluators following standardized protocols, the best prac- 
tice in the field. Finally, children were assessed before and 
after intervention, allowing preintervention functioning 
(a strong predictor of post- and follow-up functioning) to 
be controlled in analyses. 


Implications for School Nursing Practice 


School nurses are able to use evidence-based practices to 
reduce children’s anxiety, which is one of the most prevalent 
mental health concerns in schools. The current study sug- 
gests that school nurses should be encouraged to seek train- 
ing in evidence-based practices for student mental health 
and for anxiety in particular. Once trained, nurses should 
strive to implement interventions with adherence to the pro- 
tocol, which will promote better outcomes for students. 
Nurses are also encouraged to use any ongoing implemen- 
tation supports after training, such as supervision, consulta- 
tion, or a support network of other nurses, as these supports 
may increase self-efficacy, which is linked to better out- 
comes for children in more complex interventions. Finally, 
nurses who are implementing simple, straightforward inter- 
ventions are encouraged to persist even if they do not feel 
confident in their ability, as our results suggest that nurses’ 
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self-efficacy may not be linked to children’s outcomes when 
interventions are not complex. These recommendations 
would be reinforced by replication of the current results in 
larger samples using more psychometrically sound 
measures. 


Conclusions 


In summary, the current study reinforces the importance of 
research on factors that affect implementation and treatment 
outcomes. Modifiable implementation factors including pro- 
vider adherence and self-efficacy are important targets for 
implementation processes like training and consultation. 
Intervention complexity is a factor that should be considered 
in the intervention selection and adaptation phases of imple- 
mentation. The current study suggests that by understanding 
and addressing these implementation factors, client out- 
comes can be improved. 


Authors’ Note 
The opinions expressed are those of the authors and do not repre- 


sent views of the Institute of Education Sciences or the U.S. 
Department of Education. 


Acknowledgments 


The authors thank the CALM Development Team for their expert 
guidance and feedback throughout the course of the project: Donna 
Mazcyk, Nichole Bobo, Dr. Mark Weist, Dr. Catherine Bradshaw, 
and Stephanie Knutson. The authors also thank all the nurses, chil- 
dren, and parents who participated in the study. 


Author Contributions 


Kelly L. Drake and Golda S. Ginsburg contributed to the design of 
the research study, while the draft was prepared by EB Caron and 
Golda S. Ginsburg. All authors contributed to the acquisition, anal- 
ysis, and interpretation of data; revisions of the article; gave final 
approval; and agreed to be accountable for all aspects of work 
ensuring integrity and accuracy. 


Declaration of Conflicting Interests 


The author(s) declared no potential conflicts of interest with respect 
to the research, authorship, and/or publication of this article. 


Funding 


The author(s) disclosed receipt of the following financial support 
for the research, authorship, and/or publication of this article: This 
research was funded by the Institute of Education Sciences, U.S. 
Department of Education Grant #R305A140694 awarded to Drs. 
Ginsburg and Drake. 


ORCID iD 
EB Caron, PhD © https://orcid.org/0000-0002-3717-2979 


References 


Becker, E. M., Becker, K. D., & Ginsburg, G. S. (2012). Modular 
cognitive behavioral therapy for youth with anxiety disorders: A 
closer look at the use of specific modules and their relation to 


treatment process and response. School Mental Health, 4, 
243-253. https://doi.org/10.1007/s123 10-012-9080-2 

Bjaastad, J. F., Wergeland, G. J. H., Haugland, B. S. M., Gjestad, R., 
Havik, O. E., Heiervang, E. R., & Ost, L. G. (2018). Do clin- 
ical experience, formal cognitive behavioural therapy training, 
adherence, and competence predict outcome in cognitive 
behavioural therapy for anxiety disorders in youth? Clinical 
Psychology & Psychotherapy, 25, 865-877. https://doi.org/10. 
1002/cpp.2321 

Bohman, B., Ghaderi, A., & Rasmussen, F. (2014). Training in 
methods of preventing childhood obesity increases self- 
efficacy in nurses in child health services: A randomized, con- 
trolled trial. Journal of Nutrition Education and Behavior, 46, 
215-218. https://doi.org/10.1016/j.jneb.2013.10.006 

Breitenstein, S. M., Gross, D., Garvey, C. A., Hill, C., Fogg, L., & 
Resnick, B. (2010). Implementation fidelity in community- 
based interventions. Research in Nursing & Health, 33, 
164-173. https://doi.org/10.1002/nur.20373 

Campbell, B. K., Buti, A., Fussell, H. E., Srikanth, P., McCarty, D., 
& Guydish, J. R. (2013). Therapist predictors of treatment 
delivery fidelity in a community-based trial of 12-step facilita- 
tion. The American Journal of Drug and Alcohol Abuse, 39, 
304-311. https://doi.org/10.3109/00952990.2013.799175 

Chavira, D. A., Stein, M. B., Bailey, K., & Stein, M. T. (2004). 
Child anxiety in primary care: Prevalent but untreated. 
Depression and Anxiety, 20, 155-164. https://doi.org/10. 
1002/da.20039 

Copeland, W. E., Angold, A., Shanahan, L., & Costello, E. J. 
(2014). Longitudinal patterns of anxiety from childhood to 
adulthood: The Great Smoky Mountains Study. Journal of the 
American Academy of Child & Adolescent Psychiatry, 53, 
21-33. https://doi.org/10.1016/j.jaac.2013.09.017 

Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alex- 
ander, J. A., & Lowery, J. C. (2009). Fostering implementation 
of health services research findings into practice: A consoli- 
dated framework for advancing implementation science. 
Implementation Science, 4, 50. https://doi.org/10.1186/1748- 
5908-4-50 

David, P., & Schiff, M. (2017). Self-efficacy as a mediator in 
bottom-up dissemination of a research-supported intervention 
for young, traumatized children and their families. Journal of 
Evidence-Informed Social Work, 14, 53-69. https://doi.org/10. 
1080/23761407.2017.1298072 

Drake, K. L., Stewart, C. E., Muggeo, M. A., & Ginsburg, G. S. 
(2015). Enhancing the capacity of school nurses to reduce 
excessive anxiety in children: Development of the CALM inter- 
vention. Journal of Child and Adolescent Psychiatric Nursing, 
28, 121-130. https://doi.org/10.1111/jcap.12115 

Edmunds, J. M., Beidas, R. S., & Kendall, P. C. (2013). Dissemi- 
nation and implementation of evidence-based practices: Train- 
ing and consultation as implementation strategies. Clinical 
Psychology: Science and Practice, 20, 152-165. https://doi. 
org/10.1111/cpsp.12031 

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & 
Wallace, F. (2005). Implementation research: A synthesis of the 


Caron et al. 


literature (FMHI Publication No. 231). University of South 
Florida, Louis de la Parte Florida Mental Health Institute, The 
National Implementation Research Network. https://fpg.unc. 
edu/node/4445 

Foster, S., Rollefson, M., Doksum, T., Noonan, D., Robinson, G., 
& Teich, J. (2005). School mental health services in the United 
States, 2002-2003 (DHHS Publication No. (SMA) 05-4068). 
Center for Mental Health Services, Substance Abuse and Men- 
tal Health Services Administration. https://files.eric.ed.gov/full 
text/ED499056.pdf 

Ginsburg, G. S., Becker, K. D., Drazdowski, T. K., & Tein, J. Y. 
(2012). Treating anxiety disorders in inner city schools: Results 
from a pilot randomized controlled trial comparing CBT and 
usual care. Child & Youth Care Forum, 41, 1-19. https://doi. 
org/10.1007/s10566-011-9156-4 

Ginsburg, G. S., Drake, K. L., Muggeo, M. A., Stewart, C. E., 
Pikulski, P. J., Zheng, D., & Harel, O. (2019). A pilot RCT of 
a school nurse delivered intervention to reduce student anxiety. 
Journal of Clinical Child & Adolescent Psychology. https://doi. 
org/10.1080/15374416.2019.1630833 

Green, B., Shirk, S., Hanze, D., & Wanstrath, J. (1994). The Chil- 
dren’s Global Assessment Scale in clinical practice: An empiri- 
cal evaluation. Journal of the American Academy of Child & 
Adolescent Psychiatry, 33, 1158-1164. https://doi.org/10.1097/ 
00004583-199410000-0001 1 

Grol, R. P., Bosch, M. C., Hulscher, M. E., Eccles, M. P., & Wen- 
sing, M. (2007). Planning and studying improvement in patient 
care: The use of theoretical perspectives. The Milbank Quar- 
terly, 85, 93-138. https://doi.org/10.1111/).1468-0009.2007. 
00478.x 

Guy, W. (1976). ECDEU Assessment Manual for Psychopharma- 
cology—Revised (DHEW Publication No. ADM 76-338). 
National Institute of Mental Health. 

Hoeft, T. J., Fortney, J. C., Patel, V., & Uniitzer, J. (2018). Task- 
sharing approaches to improve mental health care in rural and 
other low-resource settings: A systematic review. The Journal 
of Rural Health, 34, 48-62. https://doi.org/10.1111/jrh.12229 

Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to 
the field: The role of fidelity and achieved relative intervention 
strength. Journal of Research on Educational Effectiveness, 2, 
88-110. https://doi.org/10.1080/19345740802539325 

James, A. C., James, G., Cowdrey, F. A., Soler, A., & Choke, A. 
(2015). Cognitive behavioural therapy for anxiety disorders in 
children and adolescents. Cochrane Database of Systematic 
Reviews. https://www.cochranelibrary.com/cdsr/doi/10.1002/ 
14651858.CD004690.pub4/full. 

Klimes-Dougan, B., August, G. J., Lee, C. Y. S., Realmuto, G. M., 
Bloomquist, M. L., Horowitz, J. L., & Eisenberg, T. L. (2009). 
Practitioner and site characteristics that relate to fidelity of 
implementation: The Early Risers prevention program in a 
going-to-scale intervention trial. Professional Psychology: 
Research and Practice, 40, 467-475. https://doi.org/10.1037/ 
a0014623 

Lang, C., Nir, Z., Gothelf, A., Domachevsky, S., Ginton, L., Kush- 
nir, J., & Gothelf, D. (2016). The outcome of children with 


selective mutism following cognitive behavioral intervention: 
A follow-up study. European Journal of Pediatrics, 175, 
481-487. https://doi.org/10.1007/s0043 1-015-2651-0 

Liber, J. M., McLeod, B. D., Van Widenfelt, B. M., Goedhart, A. 
W., van der Leeden, A. J., Utens, E. M., & Treffers, P. D. 
(2010). Examining the relation between the therapeutic alliance, 
treatment adherence, and outcome of cognitive behavioral ther- 
apy for children with anxiety disorders. Behavior Therapy, 41, 
172-186. https://doi.org/10.1016/j.beth.2009.02.003 

Merikangas, K. R., He, J. P., Brody, D., Fisher, P. W., Bourdon, K., 
& Koretz, D. S. (2010). Prevalence and treatment of mental 
disorders among US children in the 2001-2004 NHANES. 
Pediatrics, 125, 75-81. https://doi.org/10.1542/peds.2008-2598 

Merikangas, K. R., He, J. P., Burstein, M., Swendsen, J., Avene- 
voli, S., Case, B., Georgiades, K., Heaton, L., Swanson, S., & 
Olfson, M. (2011). Service utilization for lifetime mental dis- 
orders in US adolescents: Results of the National Comorbidity 
Survey—Adolescent Supplement (NCS-A). Journal of the Amer- 
ican Academy of Child & Adolescent Psychiatry, 50, 32-45. 
https://doi.org/10.1016/j.jaac.2010.10.006 

Mufson, L., Dorta, K. P., Wickramaratne, P., Nomura, Y., Olfson, 
M., & Weissman, M. M. (2004). A randomized effectiveness 
trial of interpersonal psychotherapy for depressed adolescents. 
Archives of General Psychiatry, 61, 577-584. https://doi.org/ 
10.1001/archpsyc.61.6.577 

Muggeo, M. A., & Ginsburg, G. S. (2019). School nurse percep- 
tions of student anxiety. The Journal of School Nursing, 35, 
163-168. https://doi.org/10.1177/10598405 17752457 

Muggeo, M. A., Stewart, C. E., Drake, K. L., & Ginsburg, G. S. 
(2017). A school nurse-delivered intervention for anxious chil- 
dren: An open trial. School Mental Health, 9(2), 157-171. 
https://doi.org/10.1007/s12310-017-9211-x 

Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity 
and therapeutic change: Issues and research recommendations. 
Clinical Psychology: Science and Practice, 12, 365-383. 
https://doi.org/10.1093/clipsy.bpi045 

Podell, J. L., Kendall, P. C., Gosch, E. A., Compton, S. N., March, 
J.S., Albano, A. M., Rynn, M. A., Walkup, J. T., Sherrill, J. T., 
Ginsburg, G. S., Keeton, C. P., Birmaher, B., & Piacentini, J. C. 
(2013). Therapist factors and outcomes in CBT for anxiety in 
youth. Professional Psychology: Research and Practice, 44, 
89-98. https://doi.org/10.1037/a003 1700 

Salloum, A., Johnco, C., Lewin, A. B., McBride, N. M., & Storch, 
E. A. (2016). Barriers to access and participation in community 
mental health treatment for anxious children. Journal of Affec- 
tive Disorders, 196, 54-61. https://doi.org/10.1016/j.jad.2016. 
02.026 

Schiele, B. E., Weist, M. D., Youngstrom, E. A., Stephan, S. H., & 
Lever, N. A. (2014). Counseling self-efficacy, quality of ser- 
vices and knowledge of evidence-based practices in school 
mental health. The Professional Counselor, 4, 467-480. 
https://doi.org/10.15241/bes.4.5.467 

Shaffer, D., Gould, M. S., Brasic, J., Ambrosini, P., Fisher, P., Bird, 
H., & Aluwahlia, S. (1983). A Children’s Global Assessment 


The Journal of School Nursing XX(X) 


Scale (CGAS). Archives of General Psychiatry, 40, 1228-1231. 
https://doi.org/10.1001/archpsyc.1983.01790100074010 

Stallard, P., Simpson, N., Anderson, S., & Goddard, M. (2008). The 
FRIENDS emotional health prevention programme. European 
Child & Adolescent Psychiatry, 17, 283-289. https://doi.org/10. 
1007/s00787-007-0665-5 

Stallard, P., Simpson, N., Anderson, S., Hibbert, S., & Osborn, C. 
(2007). The FRIENDS emotional health programme: Initial 
findings from a school-based project. Child and Adolescent 
Mental Health, 12, 32-37. https://doi.org/10.1111/).1475-3 
588.2006.00421.x 

Swan, A. J., & Kendall, P. C. (2016). Fear and missing out: Youth 
anxiety and functional outcomes. Clinical Psychology: Science 
and Practice, 23, 417-435. https://doi.org/10.1111/cpsp.12169 

Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. 
(2012). Bridging research and practice: Models for dissemina- 
tion and implementation research. American Journal of Preven- 
tive Medicine, 43, 337-350. https://doi.org/10.1016/j.amepre. 
2012.05.024 

Turner, K. M., Nicholson, J. M., & Sanders, M. R. (2011). The role 
of practitioner self-efficacy, training, program and workplace 
factors on the implementation of an evidence-based parenting 
intervention in primary care. The Journal of Primary Preven- 
tion, 32, 95-112. https://doi.org/10.1007/s10935-011-0240-1 

Walkup, J. T., Albano, A. M., Piacentini, J., Birmaher, B., 
Compton, S. N., Sherrill, J. T., Ginsburg, G. S., Rynn, M. 
A., McCracken, J., Waslick, B., Iyengar, S., March, J. S., & 
Kendall, P. C. (2008). Cognitive behavioral therapy, sertra- 
line, or a combination in childhood anxiety. New England 
Journal of Medicine, 359, 2753-2766. https://doi.org/10. 
1056/nejmoa0804633 


Weist, M. D., Bruns, E. J., Whitaker, K., Wei, Y., Kutcher, S., 
Larsen, T., Holsen, I., Cooper, J. L., Geroski, A., & Short, K. 
H. (2017). School mental health promotion and intervention: 
Experiences from four nations. School Psychology Interna- 
tional, 38, 343-362. https://doi.org/10.1177/01430343 
17695379 

Zvoch, K. (2009). Treatment fidelity in multisite evaluation a mul- 
tilevel longitudinal examination of provider adherence status 
and change. American Journal of Evaluation, 30, 44-61. 
https://doi.org/10.1177/1098214008329523 


Author Biographies 


EB Caron, PhD, is an assistant professor at the Department of 
Psychological Science, Fitchburg State University, Fitchburg, MA. 


Kelly L. Drake, PhD, is an assistant professor at the Department of 
Psychiatry and Behavioral Sciences, Johns Hopkins University, 
Baltimore, MD and a licensed clinical psychologist at the Anxiety 
Treatment Center of Maryland, Columbia, MD. 


Catherine E. Stewart, PhD, is a clinical psychologist who com- 
pleted this research while a student in the Department of Psycho- 
logical Sciences, University of Connecticut, Storrs, CT. 


Michela A. Muggeo, PsyD, is a licensed clinical psychologist at 
Clarus Health Alliance, Norwich, CT. 


Golda S. Ginsburg, PhD, is a professor and licensed clinical psy- 
chologist at the Department of Psychiatry, University of Connecti- 
cut School of Medicine, West Hartford, CT. 


